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ABSTRACT 

Introduction: It is fundamental that randomised 
controlled trials (RCTs) are properly conducted in 
order to reach well-supported conclusions. However, 
there is emerging evidence that RCTs are subject to 
biases which can overestimate or underestimate the 
true treatment effect, due to flaws in the study design 
characteristics of such trials. The extent to which this 
holds true in oral health RCTs, which have some 
unique design characteristics compared to RCTs in 
other health fields, is unclear. As such, we aim to 
examine the empirical evidence quantifying the extent 
of bias associated with methodological and non- 
methodological characteristics in oral health RCTs. 
Methods and analysis: We plan to perform a meta- 
epidemiological study, where a sample size of 60 
meta-analyses (MAs) including approximately 600 
RCTs will be selected. The MAs will be randomly 
obtained from the Oral Health Database of Systematic 
Reviews using a random number table; and will be 
considered for inclusion if they include a minimum of 
five RCTs, and examine a therapeutic intervention 
related to one of the recognised dental specialties. 
RCTs identified in selected MAs will be subsequently 
included if their study design includes a comparison 
between an intervention group and a placebo group or 
another intervention group. Data will be extracted from 
selected trials included in MAs based on a number of 
methodological and non-methodological 
characteristics. Moreover, the risk of bias will be 
assessed using the Cochrane Risk of Bias tool. Effect 
size estimates and measures of variability for the main 
outcome will be extracted from each RCT included in 
selected MAs, and a two-level analysis will be 
conducted using a meta-meta-analytic approach 
with a random effects model to allow for intra-MA 
and inter-MA heterogeneity. 
Ethics and dissemination: The intended audiences 
of the findings will include dental clinicians, oral health 
researchers, policymakers and graduate students. The 
aforementioned will be introduced to the findings 
through workshops, seminars, round table discussions 
and targeted individual meetings. Other opportunities 



Strengths and limitations of this study 



■ This meta-epidemiological study is the first of its 
kind conducted in oral health research, and will 
most likely have important implications for oral 
health research, dental practice decision-making 
and oral health policy. 

■ This work will have an important impact on the 
quality of future oral health RCTs and systematic 
reviews by providing new and important insights 
about potential biases that exist in RCTs as well 
as factors associated with bias in oral health 
RCTs. 

■ It will provide a research framework when 
appraising, reporting and conducting oral health 
RCTs. 



for knowledge transfer will be pursued such as key 
dental conferences. Finally, the results will be 
published as a scientific report in a dental 
peer-reviewed journal. 



INTRODUCTION 

Current scientific knowledge for clinical 
research is based on randomised control 
trials (RCTs) that have been synthesised in 
systematic reviews (SRs) and meta-analyses 
(MAs), which together comprise the 'gold 
standard' of scientific evidence. 1 2 The abun- 
dance of RCTs is continually increasing with 
about 50 new clinical trials published every 
month in the field of dentistry. 3 Since these 
sources are considered the highest level of 
evidence for the efficacy of treatment inter- 
ventions, the information gathered from 
them is used to guide clinical practice and 
policy decisions. 4 In the field of oral health 
research, several investigations have assessed 
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the methodological/reporting quality of oral health 
RCTs, 5-8 and examined important aspects related to the 
conduct and reporting of these trials such as: clustering 
effects, 9 reporting statistical findings, 10 sample size 
justification 11 and randomisation process. 12 13 
Nonetheless, the quality of these forms of evidence has 
not yet been fully scrutinised to identify design flaws and 
their impact on treatment effect estimates in the field of 
oral health research. 3 14 

Ideally, RCTs should be properly conducted and 
accurately reported in order to reach well-supported 
conclusions for decision-making that are both valid and 
generalisable to patients who will receive the interven- 
tions in clinical practice. 4 15 However, evidence is emer- 
ging that some RCTs are biased and overestimate the 
magnitude of the effect size due to flaws in their design 
and/or reported study characteristics. 15 16 These studies 
will likely skew the overall conclusions of MAs once 
pooled, possibly leading to faulty treatment decisions. 2 17 
Generally, the poor methodological quality of these 
RCTs has resulted in the tendency to exaggerate or over- 
estimate the true treatment effect (effect size). 18 Among 
the flawed characteristics observed in previously pub- 
lished reports and found to have an impact on the true 
treatment effect were: the lack of randomisation and 
concealment, 19 20 inadequate blinding 16 20 21 and indus- 
trial funding, 22 although not all the studies confirmed 
these associations. 23 24 This could lead clinicians to the 
implementation of treatment choices, which might be 
inappropriate or ineffective and have negative effects on 
treatment outcomes in dental practices. 3 

The assumption of the association between trial 
quality and true effect size is derived from published 
'meta-epidemiological' studies, which are investigations 
that quantify the extent of bias in the effect sizes related 
to trial quality in a group of meta-analyses. 25 There are a 
few meta-epidemiological investigations that have been 
conducted in the field of medicine ; however, 

the value of the conclusions of some of these investiga- 
tions to other healthcare fields is limited by numerous 
factors including: the examination of quality items that 
were related to reporting quality and not methodo- 
logical quality or bias, 23 25 26 the failure to examine con- 
tinuous outcomes (which occurred in the majority of 
studies) 16 23 25 based on a preference for evaluating 
dichotomous outcomes which could limit applying their 
conclusions to RCTs with continuous outcomes, and the 
presentation of inconsistent findings regarding the meth- 
odological criteria that are associated with effect size. 16 23 

More importantly, meta-epidemiological studies 
reported that bias in effect size associated with methodo- 
logical characteristics may vary between medical fields or 
different areas of health research 27 and differ based on 
the type of intervention. 20 25 To our current knowledge, 
no meta-epidemiological study has been conducted in 
any of the nine specialised dental fields that examined 
the extent of bias related to the quality of oral health 
RCTs. It is unclear to what extent this holds true in 



oral health RCTs, which have some unique design 
characteristics compared with RCTs in other health 
fields, such as: difficulty in applying blinding, use of a 
broad range of different interventions (surgical, non- 
surgical, drug and non-drug), 28 use of multiple out- 
comes, common use of split-mouth and crossover 
designs and clustering effects, 9 29 which make the evalu- 
ation of these trials more challenging compared to trials 
in other health areas. 

As such, the purpose of the proposed study is to 
provide a first step in the development of a research 
framework for appraising, reporting and conducting 
RCTs in oral health research. The objectives are to: (1) 
examine the empirical evidence for associations between 
methodological trial characteristics (eg, adequacy of ran- 
domisation, adequacy of allocation concealment, base- 
line comparability, blinding of assessors and participants, 
similarity of cointerventions, adequacy of compliance to 
the treatment, among others) and treatment effect esti- 
mates (effect sizes) in oral health RCTs and (2) determine 
if other non-methodological study characteristics (eg, the 
nature of intervention, specialty, type of outcomes, 
number of centres, type of funding, among others) are 
associated with effect sizes in oral health RCTs. 

The hypothesis of the proposed study is that there is 
no difference between treatment effect estimates (effect 
sizes) for oral health trials meeting certain methodo- 
logical quality characteristics versus trials not meeting 
those quality characteristics, such as: adequacy of ran- 
domisation, adequacy of allocation concealment, base- 
line comparability, blinding of assessors and participants, 
similarity of cointerventions, similarity of outcome assess- 
ment, description of withdrawals and adequacy of com- 
pliance to the treatment, among others; and that trials 
with different non-methodological characteristics such as 
nature of intervention, type of outcomes, study design, 
number of centres, type of funding, sample size and spe- 
ciality, among others, will not have different treatment 
effect estimates (effect sizes). 



METHODS AND ANALYSIS 
Design 

A meta-epidemiological study. 

Study selection 

Selection of MAs 

MAs/SRs will be included if they fulfil the following 
inclusion criteria: 

1. Reports should be therapeutic oral health MAs 
defined as reviews that examined therapeutic inter- 
ventions related to dental/ oral diseases as defined by 
the American Dental Association (ADA) scope of 
practice. 28 30 Reports will be considered as MAs if 
they explicitly identified and summarised evidence 
from several published reports through quantitative 
analyses. 31 32 
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2. MAs should include a minimum of five RCTs and 
provide quantitative data of treatment estimates. 

3. The MA should examine at least one continuous 
outcome. 

4. MAs should be full-length reports. 

Selection of RCTs 

All RCTs included in selected MAs will be eligible if they 
meet the following inclusion-exclusion criteria: 

1 . The study design is reported to be an RCT; 33 

2. Comparison is between an intervention and a 
placebo or another intervention; 

3. RCTs evaluated a therapeutic intervention related to 
one of the dental specialties defined by the ADA; 30 

4. RCTs will be excluded if the results are reported in a 
way that does not allow for effect sizes to be calculated. 

Sample size calculation 

Previously published meta-epidemiological studies have 
been reported to be underpowered because of their 
small sample sizes and their highly heterogeneous 
samples. 27 Accordingly, it has been suggested compiling 
a set of RCTs that are specific to clinical fields to 
decrease heterogeneity and improve power of meta- 
epidemiological reports. Our study will focus on oral 
health RCTs published in the recognised nine dental 
specialties. Furthermore, a minimum of 60 MAs contain- 
ing approximately 600 RCTs will be assessed for this 
meta-epidemiological study. Given the previous 
reports, 34 35 it could be anticipated that a difference in 



effect sizes of at least 0.15 will be obtained between trials 
with and without selected quality domains. This magni- 
tude of difference has been argued to correspond to one- 
quarter to one-half of the typical treatment effect found 
for interventions in areas similar to dentistry. Thus, this 
difference should also be relevant to the field of dentistry. 

As such, we planned a sample size of 60 MAs expect- 
ing that a sample of 600 RCTs would come from these 
MAs. To the best of our knowledge, this number of 
RCTs will represent the largest sample size examined in 
a meta-epidemiological study aimed at examining bias 
related to trial quality using continuous outcomes. The 
sample of MAs will be selected from the Oral Health 
Database of Systematic Reviews, 28 developed by the 
authors and include all of the oral health SRs published 
between 1999 and 2012, encompassing the nine dental 
specialties defined by the ADA. 30 This database contains 
153 MAs (39 Cochrane and 114 non-Cochrane), out of 
the 1188 SRs included in the database, which potentially 
meet the inclusion-exclusion criteria for this study. 
Figure 1 provides further details of the SRs identified in 
the database and the number of MAs potentially 
meeting the eligibility criteria for this study, within each 
of the nine dental specialties and for Cochrane and 
non-Cochrane SRs separately. 

Data extraction 

A data extraction template will be designed in Microsoft 
ACCESS and pilot tested. With regard to assessors, a 
panel of assessors from varied health research 



SRs included in the Oral Health Database of 
Systematic Reviews 28 
(1188 SRs: 126 Cochrane & 1062 non-Cochrane) 



Dental Public 
Health 

(184 SRs: 
NCRs=163 & 
CRs=21) 



(212 SRs: 
NCRs=203 & 
CRs=9) 



Oral 
Surgery & 
Implantology 

(159 SRs: 
NCRs=134& 
CRs=25) 



Oral 
Medicine & 
Pathology 

(162 SRs: 
NCRs=140 & 
CRs=22) 



Prosthodontics 
& Restorative 
Dentistry 

(198 SRs: 
NCRs=179 & 
CRs=19) 



Orthodontics 
& Den tola rial 
Orthopedics 

(138 SRs: 
NCRs=123 & 
CRs=15) 



(54 SRs: 
NCRs=47 & 
CRs=7) 



Pediatric 
Dentistry 

(50 SRs: 
NCRs=42 & 
CRs=8) 



Oral and 
Radiology 

(31 SRs: 
NCRs=31 & 
CRs=0) 



36 MAs 

potentially met 
the eligibility 
criteria (21 
NCRs & 15 
CRs) 



43 MAs 

potentially met 
the eligibility 
criteria (40 
NCRs & 3 
CRs) 



29 MAs 

potentially met 
the eligibility 
criteria (23 
NCRs & 6 
CRs) 



17 MAs 

potentially met 
the eligibility 
criteria (5 
NCRs & 12 
CRs) 



10 MAs 

potentially met 
the eligibility 
criteria (9 
NCRs & 1 
CRs) 



7 MAs 

potentially met 
the eligibility 
criteria (7 
NCRs&O 
CRs) 



SMAs 

potentially met 
the eligibility 
criteria (7 
NCRs & 1 
CRs) 



3 MAs 
potentially met 
the eligibility 
criteria (2 
NCRs & 1 
CRs) 



OMAs 

potentially met 
the eligibility 
criteria (0 
NCRs & 0 
CRs) 



A minimum of 600 RCTs included in 60 
MAs will be obtained for this meta- 
epidemiological study 



Figure 1 Systematic reviews identified in the database and the number of MAs potentially meeting the eligibility criteria for this 
study. 
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backgrounds will perform data extraction. One of the 
team members will perform the training for all assessors 
and will make sure that all of them have a clear under- 
standing of the data extraction process. Training of 
these assessors will be carried out through 10-15 separ- 
ate articles, not included in the set of articles to be 
reviewed. Each of the 10-15 training articles will be 
independently reviewed by all the members of the 
review panel and then discussed by the panel. In order 
to ensure good agreement between the assessors, the 
training exercise will be repeated to address any issues 
identified in the first exercise. For actual data extraction, 
two assessors will independently complete data extrac- 
tion with a consensus meeting utilised to resolve any dis- 
agreement between the assessors. If a consensus could 
not be achieved, then the two assessors will consult with 
a third assessor (HS or SA-O) to achieve full consensus, 
and only consensus answers will be used for all analyses. 
Data will be extracted on the following items. 

Non-methodological characteristics 

Dental speciality (eg, dental public health, endodontics, 
oral medicine and oral pathology, oral and maxillofacial 
radiology, oral and maxillofacial surgery, orthodontics 
and dentofacial orthopaedics, pediatric dentistry, peri- 
odontics, prosthodontics and restorative dentistry), year 
of publication, source of funding (eg, industry, govern- 
ment, foundation and academic), type of intervention 
(eg, drug, surgical, device, dental material, psycho- 
logical, educational and policy), number of randomised 
groups, number of centres (eg, multicentre and single 
centre), study design (eg, parallel, crossover and factor- 
ial) and type of outcome (eg, subjective and objective). 

Methodological characteristics 

Methodological characteristics will be based on preliminary 
work performed by our research team and will be 
extracted from commonly used tools to evaluate the 
methodological quality of RCT in health research, 36 such 
as: adequacy of randomisation, adequacy of allocation 
concealment, baseline comparability, blinding of asses- 
sors and participants, similarity of cointerventions, simi- 
larity of outcome assessment, description of withdrawals 
and adequacy of compliance to the treatment. 
Guidelines for decision-making will be formed based on 
the previous work of our team, in order to increase 

36 37 

consistency. 

Risk of bias 

Risk of bias will be assessed using the Cochrane risk of 
bias tool as 'high', 'low' or 'unclear' and will follow the 
guidelines of the Cochrane Collaboration for scoring 
clinical trials. 33 

Treatment effect estimates 

Treatment effect estimates, measures of variability (SDs 
and 95% CIs) and respective sample sizes will be 
extracted for the main outcome. 



Data analysis 

Descriptive statistics (means and SDs) will be extracted 
for one continuous outcome per RCT. RCTs that are 
included in more than one MA will be eligible for inclu- 
sion in the study only once, and will be extracted from 
the MA with the fewer number of RCTs. STATA statistical 
software V12 will be employed to perform the planned 
statistical analyses. 

We propose to use a two-level meta-meta-analytic 
approach with the use of a random effects model. This 
analysis will permit the evaluation of the heterogeneity 
intra-MAs and inter-MAs. 38 The first step will consist in 
obtaining the standardised effect size estimates for the 
primary outcome of each trial using the guidelines 
established by Cohen. 39 The second step will involve 
pooling the results of the previous analysis, using a com- 
bined difference approach, to demonstrate the different 
components of MAs across all MAs. Moreover, the data 
acquired will be used to evaluate certain components of 
the methodological assessment — such as allocation con- 
cealment, randomisation, blinding, etc — and will subse- 
quently be used to divide the data set into two groups: 
one group having adequately addressed the said compo- 
nent, and the other not addressing it. Thus, for each 
MA, we will conduct metaregression techniques to 
derive the difference between pooled estimates from 
trials with (eg, allocation concealment) and without the 
characteristic of interest. Formal tests of interaction will 
be performed separately for each MA based on z scores 
for estimated differences in effect sizes between trials 
with and without the characteristic of interest and the 
corresponding SEs. Therefore, two pooled effect sizes 
will be calculated for each MA. The effect sizes at this 
stage will be combined using the DerSimonian and 
Laird random effect models to allow for appropriate 
inter-MA heterogeneity assessment. 40 P values will be 
two sided. Analysis will be performed by a statistician 
specialised in the meta-epidemiological approach. 



DISCUSSION 

The findings of the proposed research could most likely 
have important implications for oral health research, 
dental practice decision-making and oral health 
policy. The proposed research will be the first meta- 
epidemiological study that provides empirical evidence 
regarding biases related to the quality of RCTs in the 
field of dentistry. This work, in combination with some 
of the current knowledge the oral health community 
already has, should have an important impact on the 
quality of future oral health RCTs, SRs and MAs by pro- 
viding new and important insights about potential biases 
that exist in RCTs as well as factors associated with bias 
in oral health RCTs. Additionally, it will provide an 
improved framework when conducting, appraising and 
reporting oral health RCTs. 

More importantly, this additional information will 
update dental professionals about proper, evidence- 
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based decision-making when treating patients and could 
assist guideline developers and policymakers in making 
informed decisions about the implementation of dental 
interventions. Finally, the outcomes generated from this 
work should most likely be of value for developing and 
disseminating future research framework for the 
conduct and reporting of oral health RCTs. 

Dissemination of the developed framework will be 
achieved through an array of means to maximise expos- 
ure. The intended audiences will include dental clini- 
cians, oral health researchers, policymakers and 
graduate students. The aforementioned persons will be 
introduced to this research framework through work- 
shops, seminars, round table discussions and targeted 
individual meetings. Moreover, key organisations will be 
used to strengthen the dissemination strategy, such as 
the International Association for Dental Research 
(IADR), the American Dental Association (ADA), the 
Canadian Dental Association (CD A) and the Cochrane 
Bias Methods Group. Other opportunities for knowledge 
transfer will be pursued such as key conferences (eg, the 
annual meeting of the International Association for 
Dental Research) . Finally, the results of this study will be 
published as a scientific report in a dental peer-reviewed 
journal. 
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